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Simulates, integrates, generates and learns using spatiotemporal data 

1. (Simulated) Drones (fly around in space and time) 

2. Cadence + Kafka = integration across different Timescales (Slow and Fast) Watch out for the 
Geospatial highlights 


3. Drone simulation generates lots of spatiotemporal data 


4. Incremental Machine Learning over spatiotemporal drifting data is tricky 


5. We live in a 4 (at least) dimensional spacetime universe © 
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"Henceforth space by itself and time by itself are doomed to fade away into mere 


shadows, and only a kind of union of the two will preserve an independent reality" — Special and general relativity 

Minkowski, 1908 Minkowski Light Cone (for causality, 
showing the future, the past, and 
"elsewhere") (Wikipedia) 


Cloud Platform for Big 
Data Open Source 
Technologies 


A recent addition is 
Workflow Orchestration 
with Uber's Open Source 
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Workflow Orchestration (Simple 


Concept): Task Ordering 
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Workflow Orchestration (Simple 
Concept): Task Ordering 
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Workflow Orchestration (Simple 


Concept): Task Ordering 


But Harder In Practice... 


Workload Orchestration with (Cadence 


Pedalling with a high Cadence (pedal revolutions) is called Spinning! 
Mashing or grinding is slow (and bad). ——— 


(Source: Shutterstock) 
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m Cadence is Fault-Tolerant—Workflows can fail! a c 
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Cadence supports Long Running & Scheduled Workflows 


(Source: Photo by Soly Moses from Pexels) 
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@Override 

public void startWorkflow(String a) { 
String b = activities.taskl (a); 
String c = activities.task2(b); 


A workflow implementation with 2 tasks executed sequentially 


Cadence Architecture 


Instaclustr Managed Service 


Customer Managed Instaclustr Managed 
Clients and Workers Cadence and Database Clusters 


Workers Cadence Servers 
Run workflow logic 


Java and Go Clients 
are supported 
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Database 


Cassandra 
PostgreSAL 


Kafka 
OpenSearch 
(optional) 
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Database 


State Change History Written to Database — 
Write efficient — classic time vs. space (network) tradeoff 
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Failure causes complete history 
replay 


Database 
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Replay 


Database 


Replaying workflow history recreates 
Workflow state and restart point — 
Read inefficient but hopefully infrequent 


Database 


S0 
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Replay 


Database 


= S0 +51 + 62 


Workflow state is recreated when 
tasks are scheduled to start 


Activities (including 
backwards cycling) 
can fail 
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Cadence Activities | | 


* Activities are core to Cadence, they implement tasks 

* Remote calls can fail, so wrap them in Activities 

* They can contain any code 

= Activities are executed at most once and can be automatically retried on failure 


= But must be deterministic 


Activity: Task 1 Activity: Task 2 
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Good Use Cases? | 


= 100s to millions of running workflows 

* Longrunning workflows, sleeping and scheduled tasks 

= Stateful fault-tolerant applications 

* Complex workflows 

= Integration with unreliable external systems 

= Integration with streaming and event-based systems (e.g. Kafka) 


= Short workflows 
= Short workflows (100s of steps) 
= Long workflows (1000s of steps) may take longer to replay history 


For example? Financial, retail, delivery! {Cadence 
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Charged ready to go at base 

Get an Order to deliver 

Fly to Order location and collect Order 
Fly to Delivery location and drop Order 
Fly back to base 

Recharge 


Repeat 


Return 
To Base 


Customer 


Travel To 
Destination 
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Example (Pirates) of Drone Delivery ipei 


Flight 


3 Legs: Base to Order to 
Delivery to Base 
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Returning to Base Example 


Drone flight path is computed in an 
activity 


Using location, distance, bearing, 
speed, and charge 


Every 10 seconds 


On failure, the drone won't crash 
and will continue flying from 
the last location 


(Source: Shutterstock) 
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Orders Are Also Stateful— Workflow 


2nd Workflow: Order Workflow 


Start, Generate locations, ready for drone, update locations, End if delivered 
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So, we need coordination 
between the workflows: 


(1) Asynchronous Signaling 
between workflows 


(Source: https://bahumbug.wordpress.com/2014/08/03/kafka-onna-bike/) 
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(1) Starting a workflow 
from Kafka 


ee 


kafka. 


(Cadence 


Kafka: 
New Order 


Kafka: 


Kafka: 
Drone Ready | Order Ready 


Cadence: 
Order Workflow 


Cadence: 
Drone Workflow 


Create New Order Start New Order Workflow 


Kafka 
Producer 
Allocate Order 


Order Ready 
To Drone 
Orders Ready 


Kafka 
Producer 
Order ID Kafka 


Topic 
Consumer 


New Orders Kafka 
Topic Consumer 


Drone Ready 
Kafka Drone Ready 
Producer Topic 
Drone ID 


Generate i 
Order and : Ready Update T 
Delivery | for Drone e» State and + 
| Locations i Pickup Location 
ES ; Activity: Wait for Delivered? End 
ee i Kafka Producer Signal Workflow 
i [ Start New 
Workflow 
/ Readv For nextLeg nextLeg nextLeg 
( ) orde e» (to order (to delivery —>| (return to 
d N location) location) base) 
Start Wait for 
Workflow Activity: signal Activity Activity Activity 


Kafka Producer 


(2) Using a Kafka microservice 
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Synchronous Events Asynchronous Events 

Stateful flows Stateless events 

Sequences One-off events 

Slow/long-running flows Fast/instantaneous events 
Sleep/Schedule events Real-time processing of events 
Complex flow logic Complex Stream Processing (Kafka 


Streams) 


— How Many Drones 
Can We Fly? 
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VCPUs Per Cluster (32 total) 


mCadence mCassandra "Client (EC2) 
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2,000 Drones + 2,000 Orders = 4,000 Workflows — 7 
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Part 2: Kafka*ML (TensorFlow) 


Goal: Predict which shops 
will be busy 
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Part 2: Kafka*ML (TensorFlow) 


Goal: Predict which shops 
will be busy, or not busy 


(Source: Getty Images) 
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Big Data—lots of data 


Fast Data—streaming, new data 
constantly arriving 


Changing Data—concept drift 


Just-in-time learning—continuous 
learning, model must be up to date 


Streaming data is infinite, so can't 
assume access to all past data 


The Drone Delivery System 


Generates massive amounts of 
spatiotemporal data 


Drone order data sent to Kafka 


Can we learn how to predict which shops will 
be busy in an hour? 
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Sent to TensorFlow 


TensorFlow is a 
modern neural 
network framework 
which supports 
incremental learning 


6,000 samples over a week 


Class: Shop busy/not busy per hour 
Class and features are computed from 
hourly aggregated delivery/shop data 


Other features include weekday, shop 
type, average delivery time, etc. 


Time is quantized to hours, location to 
“suburb” (this is “cheating” and ignores -= 
actual time/space scale granularity) 


Rule is a combination of shop type, 
location, weekday and hour 


In step 8, every time the “fit” 
method is called it updates 
the model - this is critical for 
Incremental learning 
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In step 6, we used the “Adam' 
optimizer 


Can be misleading unless roughly equal positive/negative samples 


Training: Accuracy improved during training from 0.5 to 0.88 over 125 Epochs 
Evaluation confusion matrix — also good results 

But! Results not repeatable, accuracy (0.5-0.9) and training time (5-92s) vary 
Sensitive to batch size (number of samples used per epoch: 8-64, 32 most consistent) 
And it sometimes gets stuck in a bad local minima. 


Predicted Class 


Data size = 6,000 samples 
Shop busy/not busy 


hourly over a week 


Actual 
Class 


Training/evaluation only 
use a subset 


Goals: 


Continuous model training 

From streaming data 

Incremental learning—don't need all the data 
Model is up-to-date all the time 

Useful for real-time use cases 

Can cope with concept drift in the data 
(eventually) 


This experiment: 


Same data as last time 
But incremental learning 


O Instaclustr Pty Limited, 2023 


Incremental clicker games can be addictive 


(Source: Shutterstock) 


Approach 


Slicing the data to get 100 samples at a time 
(simulating Kafka poll) 

Fit with each sample 

Batch 8, 20 fixed epochs 

Evaluate against all previous data 


Results 


Accuracy improves with time - good 
Max accuracy comparable with Batch learning 
But Oscillation! What's going on? 


Accuracy 


Accuracy over time 


Observation Number 


m SERA LRDDUO DEY först 
ES 


d URS AINED ad 


d v a 


"Franz Kafka reading data in the style of Dali" 


Accuracy over 1 week 


But accuracy worse: 0.72 c.f. 0.9 
for batch on same data 


X-axis is just "loop" number 


Best accuracy during week 1 = 0.7 
Concept drift introduced at start of week 2 


Accuracy instantly drops to worse (0.3) 
than guessing (0.5) 


Accuracy gradually improves to 0.75 


Oscillation still 


Concept Drift - incremental training (time vs accuracy) 


same model guessing 


M "o AV Vw N A ÄN 
VU WW HA 


oS 


If accuracy drops too much, reset the model. 

Retrain using only new data — removes pre-drift model "inertia". 
Helps immediately (accuracy 0.9) 

But oscillation, and accuracy drops to 0.7 by end of week 2. 


Concept Drift - incremental training (time vs 
accuracy) 


same model reset model guessing 
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We had accidental drift! 


Even though the rules weren't changing, 
the data was changing over time. 


Why? Because temporal features (day of 
week and time of day) were used to 
determine shops busy/not busy. 


And some hours had no busy shops (end 


of day!) — so accuracy actually increased. 


Accidental Drift 


I'm late! I’m late! 
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Incremental learning may result in 
a temporal bias due to fixation on 
recent samples 


(Similar to spatial bias from nearby 
samples?) 


Model overfitting (indicated by Oscillation) on time-series data is a well-known problem 
Time-series data + incremental learning = down the rabbit-hole (somewhere strange and hard to understand!) 
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Default time encodings have “temporal discontinuities” 


E.g. Time wraps around at midnight/midday (12:00 = 00:00) 


lh A Cun 


C.f. Prime Meridian and Anti-Meridian 


International date line — not a straight line! 


Both temporal and spatial — on West side it's Today, on the 
East side it's Yesterday!) 


:85,000,000 at 0° 
Miller rical Projection 


Periodicity is common in temporal (and 
spatial?) data 


Ocean Waves have Temporal and Spatial 
Periodicity 


Surfers, Gold Coast, Australia (Source: AdobeStock) 
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Time to try "no time" 


No wild oscillations 
With reset model, big drop in accuracy initially, but slightly better accuracy by end of week 2. 


In production could continue using old model until new one is trained up again. 


Concept Drift - incremental training (time vs accuracy) 
No time features 


same model reset model guessing 
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Incremental learning is fast, it should be able to keep up with streaming data in a production 
system 


And incremental learning is able to keep the model up to date (more or less), in conjunction 
with resetting 


Model accuracy may oscillate due to the presence 
of time features, fixation on recent data, changes in sample 
class distribution over time, and concept drift 


Overfitting on timeseries data is something to watch out 
for and take action to prevent/mitigate - maybe try 
“Rolling/Sliding Windows" (better for concept drifts) 


Try multiple competing models concurrently 


Rolling maybe better | 


Potential to use Cadence Workflows for MLOps 


Simulated spatiotemporal Drone Delivery system : 
* Cadence * Kafka integrated different timescales (slow and fast) 
* Cadence Scheduled Activities for movement calculations 
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Generated lots of spatiotemporal data 
ML over spatiotemporal data 


Spatiotemporal challenges 
* Detecting and relearning temporal drift is tricky — similar problem for spatial drift? 
e Simplified precomputed aggregated spatiotemporal quantization (hour, suburb) 
e Watch out for 
* spatiotemporal encoding discontinuities, wraparounds and periodicity 
* bias and overfitting in incremental learning due to focus on the now (and here for 
spatial data) 


Observations are always 
from a single point in 
space and time—you only 
directly experience "here" 
& "now", and can only 
inspect past events (not 
future events), and space 
is "big"! 
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